Logo and Cover Credits

In the description of object shapes, humans usually performiconic gestures that coincide with speech (are coverbal). Marked by asimilarity between the gestural sign and the described object, iconic ges-tures may easily depict content difficult to describe using words alone.Though the expressive potential of iconic gestures in human-computercommunication is generally acknowledged, the development of applica-tion systems that take non-verbal modalities of natural communicationinto account has in most cases focused much more on pointing and sym-bolic gestures. In this paper, we describe how Imagistic Description Treerepresentations can be used in an application system for generating vir-tual objects with particular spatial features by way of verbal descriptionsand iconic gestures. 1 Defining Shape through Iconic Gestures Based on a comprehensive corpus of speech-gesture shape descriptions acquiredfrom an empirical study [4], the Imagistic Tree (IDT) was proposed as a rep-resentation for the semantics of multimodal shape-related expressions, to theend of algorithmically interpretive operational shape descriptions from gestureand speech input modalities. It extends an earlier approach [3], which modelsthe two factors of extent and (partly) profile information in gestures, but whichhas not included structured spatial organization of gesture and accompanyingspeech reflecting this factor.The IDT models object extent, profile, and structure, as the salient semanticelements contained in gesture and speech. The basic level of IDT reprepresen-tations are object schemata, in which each object is described by a collection ofup to three axes which represent the objects extents in one, two, or three spatialdimensions. Using different combinations of axes in an object schema, severalbasic objects can be represented, such as cubes, cylinders, etc.Structural aspects of an object are represented in an imagistic descriptionwhich can recursively embed further imagistic descriptions for object parts, toresult in a tree-like structure similar to the hierarchical structure used in theMarr and Nishihara [1] model. The complete tree describing an object including